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ABSTRACT 


Many methods of measuring and predicting operator workload have been developed that provide useful 
information in the design, evaluation, and operation of complex systems and which aid in developing 
models of human attention and performance. However, the relationships between such measures, 
imposed task demands, and measures of performance remain complex and even contradictory. It 
appears that we have ignored an important factor: people do not passively translate task demands into 
performance. Rather, they actively manage their time, resources, and effort to achieve an acceptable 
level of performance while maintaining a comfortable level of workload. While such adaptive, creative, 
and strategic behaviors are the primary reason that human operators remain an essential component of 
all advanced man-machine systems, they also result in individual differences in the way people respond 
to the same task demands and inconsistent relationships among measures. Finally, we are able to 
measure workload and performance, but interpreting such measures remains difficult; it is still not clear 
how much workload is “too much” or “too little” nor the consequences of suboptimal workload on 
system performance and the mental, physical, and emotional well-being of the human operators. The 
rationale and philosophy of a program of research developed to address these issues will be reviewed 
and contrasted to traditional methods of defining, measuring, and predicting human operator workload. 
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PREVIOUS RESEARCH GOALS 

TO EXPLAIN, QUANTIFY, AND PREDICT RELATIONSHIPS AMONG: 
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EFFECTIVENESS OF COMPUTER-GAME TRAINER IN 
IMPROVING WORKLOAD MANGEMENT SKILLS 



FLIGHT 7: LEAVING PRACTICE AREA 



PREDICTOR SCORES AFTER FLIGHT 8 

GAME TRAIWNC GROUP 



4 5 6 7 8 


PREDICTED SUCCESS SCORE 


EFFECTIVENESS OF AUTOMATION IN RELEASING 
RESOURCES TO PERFORM OTHER TASKS 
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ELEMENT 4: METHODS OF IMPROVING STRATEGIES 


MILESTONES: 


IDENTIFY OPTIMAL STRATEGIES 
FOR TYPICAL FLIGHT TASKS AND 
SITUATIONS 


DEVELOP TRAINING PROCEDURES 
TO IMPROVE PILOTS’ MANAGEMENT 
OF TIME/RESOURCES, STRATEGY 
SHIFTS APPROPRIATE FOR STATE 

DEVELOP CONCEPTUAL DESIGNS 
FOR COMPUTER AIDS TO IMPROVE 
PILOTS’ ABILITIES TO SELECT 
APPROPRIATE PLANS, STRATEGIES 
AND TACTICS 

TEST CONCEPTUAL DESIGNS FOR 
INFLIGHT ADAPTIVE SYSTEMS FOR 
DYNAMIC TASK ALLOCATION 
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INDIVIDUAL DIFFERENCES IN SUBJECTIVE WORKLOAD 

"REDLINES" 



SUBJECT 7 



SUBJECT 4 - - : 
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BOREDOM: PERFORMANCE/PHYSIOLOGICALCORRELATES 




PHYSIOLOGICAL MEASURES 


AVERAGED DATA FROM II SUBJECTS SHOWS 
CORRELATION OF 3, PHYSIOLOGICAL MEASURES 


HEART- RATE HEART-RATE HEART-RATE 

VARIABILITY VS. VARIABILITY VS. VARIABILITY VS. 

BLOCK NUMBER ALPHA VARIANCE PUPIL DIAMETER 



Block Number (5 Min/ Block} Alpha Variance (Microvolts) PupH Diameter (Mm) 



Jet Engine 
’’Fault*' Pictorial 



Computer Keypad 
Responses 


TASK PERFORMANCE 


/WER AGED DATA FROM II SUBJECTS SHOWS 
DECREMENT IN "UNDERLOAD" TASK PERFORMANCE 

MEAN REACTION TIME VS. BLOCK NUMBER 
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EFFECT OF BOREDOM ON PERFORMANCE , WORKLOAD 


INFLUENCE OF BOREDOM ON RATED WORKLOAD 



INFLUENCE OF BOREDOM ON PERFORMANCE 
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SYMPTOMS OF UNDER/OVERLOAD STATES 


WORKLOAD 


SUBJECTIVE 

EXPERIENCE: 
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ELEMENT 3: WORKLOAD "RED-LINES" 


FY89 


FY90 


FY91 


FY92 


FY93 


MILESTONES: 


IDENTIFY VARIABLES ASSOCIATED 
WITH UNDER/OVERLOAD 


IDENTIFY PERFORMANCE /PHYSIO- 
LOGICAL CORRELATES OF SUB- 
JECTIVE OVER/UNDERLOAD STATES 

INVESTIGATE ROLE OF INDIVIDUAL 
DIFFERENCES IN PERSONAL 
WORKLOAD CRITERIA 

QUANTIFY IMPACT OF STRATEGIES 
IN DYNAMIC WORKLOAD/PERFOR- 
MANCE TRADEOFFS 

MODEL WORKLOAD/PERFORMANCE 
TRADEOFFS 

QUANTIFY OVER/UNDERLOAD 
REGIONS FOR WORKLOAD MEASURES 

DEVELOP STANDARD PROCEDURES 
FOR AIRCRAFT CERTIFICATION 
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Total Response Tima (msac) 


SCHEDULING THEORY MODELS OF WORKLOAD 


SHORTEST PROCESSING TIME 


INFLUENCE OF STRATEGY ON RATED WORKLOAD 


EME AVAILABLE; 

□ 20% MORE THAN NEEDED 



SHORTEST TIME NEXT DEADLINE 


OPTIMAL STRATEGY 



EARLIEST DUE DATE DISPLAY 




TEMPORAL DYNAMICS OF MENTAL WORKLOAD 


TARGET SEQUENCE 




FROM GOAL 
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SHAPA: VERBAL/NONVERBAL PROTOCOL ANALYSIS TOOL 


SHAPA 

ENCODING SUPPORT 

REPORT GENERATION 

- FILE MANIPULATION 

- ENCODING RELIABILITY 

, - DYNAMIC CREATION OF 

- PATTERN IDENTIFICATION 

ENCODING VOCABULARY 

- DATA ANALYSES 

- PATTERN IDENTIFICATION 

- SUMMARIES 

- TEXT/ENCODING SEARCH 

i - PRINTOUT OPTIONS 

- SCREEN LAYOUT 

i J -V 




OTHER 

DATA 

ANALYSIS | 
ROUTINES 



FEATURES: 

- RUNS ON IBM-AT WITH EGA 

- FULLY INTERACTIVE 

- ENCODER DETERMINES ENCODING MODEL/THEORY 

- FASTER ENCODING 

- CHOICE OF DATA ANALYSIS TECHNIQUES 

- DIRECT ENGAGEMENT WITH DATA 


UNDER DEVELOPMENT^ MacSH AP.A 

- MULTIPLE INTERACTING AGENTS 

- MULTIPLE STREAMS OF VERBAL 

AND NON-VERBAL BEHAVIORS 

- MULTIPLE ENCODERS/RESEARCHERS 

- VISUALIZATION TOOLS 


MODEL FOR CODING VERBAL PROTOCOLS TO ASSESS 

PILOT STRATEGIES 
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WORKLOAD /PERFORMANCE FOR COMPONENT TASKS 



TRACKING ERROR FOR CONTROL TASK 



CONTROLLED AJ03 


RATED WORKLOAD OF TASK COMPONENTS 



SPATIAL VERBAL SPEED ALTTTUOE HEADING 
DISCHETE/CONTINUOUS TASK COMPONENTS 


RESPONSE LATENCY FOR DISCRETE TASKS 



SPATIAL VERBAL 

fflOCESSMQ CODE 


REAL-TIME MEASUREMENT OF MENTAL WORKLOAD 


PERCENT CORRECTLY CLASSIFIED TRIALS: 
ERP MEASURES 



ARITHMETIC TASK: RESPONSE TIME 



CONCURRENT TASK 


GAUGE MONITORING TASK: RESPONSE TIME 
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APPLICATION OF EVOKED POTENTIAL MEASURES IN 
COCKPIT SIMULATOR 


J READOUTS 
MOWTOflED 
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CHANCE IN MONITORED READOUT 
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RESPONSE TO CHANGE IN: 
NON-MONFTORED READOUT 



TIME AFTER DISPLAY CHANGE (MSEC) 


RESPONSE TO CHANGE IN: 
MONITORED READOUT 
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SENSITIVITY OF CARDIOVASCULAR MEASURES 
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INFLUENCE OF DISPLAY DESIGN ON PILOT’S HEART RATE 


STEREO vs NON-STEREO LNDG/APPR DISPLAY 
HEARTRATE INCREASE (BASELINE TO TD) 
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PILOT NUMBER 


COMPARISON AMONG MEASURES 

HEART RATE VARIABILmf 
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INFERENCES ABOUT "EFFORT" AND WORKLOAD CANNOT BE 
DRAWN FROM MEASURES OF REACTION TIME 


EXAMPLE 1: 



RESPONSE TIME WORKLOAD 



EXAMPLE 2 



RESPONSE TIME WORKLOAD 



HYPOTHETICAL RELATIONSHIPS BETWEEN TASK DEMANDS, EFFORT, 
MEASURES OF PERFORMANCE, AND WORKLOAD 


PERFORMANCE WORKLOAD 





PILOTS ADOPT DIFFERENT STRATEGIES WITHIN A FLIGHT 
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REACT DEFER 

SHED 

QUIT 


TASKS 

MISSION 

SITUATION 

; TASKS, 

: TASKS, 



UNRELATED 

TASKS 

AWARENESS, 

RELAX 

OFF-LOAD 



TO MISSION 

AHEAD OF 

REHEARSE 

PERF. 





SCHEDULE 


| CRITERIA 



PHOT 




j 



WORKLOAD; 






1 ; 

TOO 

LOW 

■ 1 ■ 



m mm 


i j 

LOW 


-■RlHl 

I,,, 

1 ■ ■ 


1 1 







? 


MODERATELY 

HIGH 


TOO 

HIGH 



HYPOTHETICAL DISTRIBUTIONS OF STRATEGIES 


CHARACTERISTICS OF STRATEGIC BEHAVIORS 



DEVELOPMENT/SETTING 
OF HIGH LEVEL GOALS 
(OPEN-LOOP) 


DYNAMIC SELECTION 
AMONG ALTERNATIVE 
SEQUENCES OF ACTIONS 
TO ACHIEVE A GOAL 


CLOSED-LOOP, RELATIVELY 
AUTOMATIC, PERFORMANCE 
OF ACTIONS APPROPRIATE 
FOR SELECTED STRATEGY 


TACTICS 



ELEMENT 2: STRATEGIC BEHAVIOR 


MILESTONES: 

DEVELOP COMMON RESEARCH 
ENVIRONMENT FOR PROGRAM 
PARTICIPANTS 

ADOPT STANDARD METHOD OF 
IDENTIFYING STRATEGIES 

QUANTIFY PERFORMANCE/WORK- 
LOAD CORRELATES OF SPECIFIC 
STRATEGIES/STRATEGY SHIFTS 

INVESTIGATE ROLE OF PILOT STATE 
AND INDIVIDUAL DIFFERENCES ON 
STRATEGIC BEHAVIOR 

CLASSIFY STRATEGIES TYPICAL OF 
VARIOUS TASKS, ENVIRONMENTS 

DETERMINE WHY PILOTS ADOPT OR 
ABANDON PLANS AND STRATEGIES 

QUANTIFY RELATIONSHIP BETWEEN 
STRATEGIES, WORKLOAD, AND 
PERFORMANCE IN FLIGHT 
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FIGURES OF MERIT - II 


GOAL: 

IDENTIFY A PARSIMONIOUS SET OF VARIABLES WHICH, IN COMBINATION, ARE 
DESCRIPTIVE OF THE INFLUENCE OF THE PILOT/VEHICLE INTERFACE DESIGN 
AND PILOT’S INTENT ON SYSTEM PERFORMANCE 

APPROACH: 

• SELECT 50 VARIABLES FROM THOSE ALREADY AVAILABLE 

• MONITOR PERFORMANCE OF NOVICE AND EXPERT PILOTS IN AFTI F-16 DURING: 

- AIR-TO-AIR MISSION 

- TERRAIN-FOLOWING MISSION 

• MEASURE PILOT WORKLOAD USING SWAT 

• SELECT PARSIMONIOUS SET OF VARIABLES USING MULTI-DIMENSIONAL SCALING, 
CLUSTER ANALYSIS, ETC 

- IDENTIFY REDUNDANT MEASURES 

- IDENTIFY MEASURES THAT PROVIDE UNIQUE INFORMATION 

- COMBINE SOME MEASURES TO CHARACTERIZE A PARTICULAR 
ASPECT OF PERFORMANCE 






TRADITIONAL MEASURES LOSE THEIR MEANING IF OPERATORS DO NOT 
TRY TO RESPOND: (1) IMMEDIATELY AND (2) PERFECTLY 

DISCRETE TASKS 


TASK 



CONTINUOUS TASKS 

DISTURBANCE 



TRADITIONAL MEASURES OF PERFORMANCE 


DISCRETE TASKS: 


TASK 



RESPONSE 





M 


MEASURES; 

MEAN REACTION TIME 
'% CORRECT 
# COMPLETED 


CONTINUOUS TASKS: 




ELEMENT 1: FIGURES OF MERIT (FoM) 


MILESTONES: 

SELECT SET OF TARGET 
TASKS 

IDENTIFY APPROPRIATE 
SUBTASK MEASURES 

SPECIFY ACCEPTABLE PER- 
FORMANCE FOR TARGET TASKS 

DEVELOP GENERALIZED 
PROCEDURES FOR CREATING 
FIGURES OF MERIT 

TEST WITH EXISTING 
DATA BASES 

USE IN LAB, SIMULATOR, FLIGHT 
RESEARCH 

INTEGRATE INTO "REDLINE" AND 
STRATEGIC BEHAVIOR ELEMENT^ 
OF PROGRAM 
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PROGRAM ORGANIZATION: LEAD ROLES 


PERFORMANCE 
CORRELATES 
(FIGURES OF MERIT) 
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PROGRAM ELEMENTS/MAJOR MILESTONES 



PRODUCTS: 

1. PREDICTIVE TOOLS FOR SYSTEM DESIGNERS 


2. STANDARD EVALUATION PROCEDURES FOR AIRCRAFT CERTIFICATION 

3. IMPROVED THEORETICAL MODEL OF WORKLOAD 

4. WORKLOAD-MANAGEMENT TRAINING CONCEPTS 

5. ADAPTIVE COMPUTER AIDS TO IMPROVE TASK ALLOCATION 


PROPOSED EXPLANATION 
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CURRENT CONCEPTUALIZATIONS OF WORKLOAD GENERALLY IGNORE 
THE DYNAMIC, ADAPTIVE, CREATIVE BEHAVIOR OF HUMAN OPERATORS 


DRIVERS RESULTING OUTCOMES 

ACTIONS 
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